Search Specification

Verse & Variable
A line of verse. A line of inquiry.
لفظ سے لہر تک — Lafz se Lehar tak

1. Purpose

This document defines the functional, editorial, technical, and accessibility requirements for the search system planned for Verse & Variable.

The publication brings together literary expressions from several languages and connects them, where appropriate, with philosophical, historical, mathematical, and scientific subjects.

Readers may approach the archive using different kinds of information.

For example, someone may search for:

The search system should support these different approaches without treating them as equivalent forms of information.

Its purpose is to make the archive discoverable while respecting linguistic accuracy, original literary expression, technical integrity, accessibility, and visitor privacy.

2. Search Design Principles

2.1 Multilingual by Design

Search should be designed around the actual writing systems and languages represented in the archive.

English should not be the only language in which meaningful searches can be performed.

2.2 Original Expression Takes Priority

Original literary text should remain independently searchable.

Search should not depend exclusively on English translations or transliterations.

2.3 Multiple Routes to Discovery

The same article may be discoverable through its title, original text, translation, author, literary form, theme, or scientific subject.

These routes should lead to the same permanent article rather than create duplicate article records.

2.4 Accurate Matching

A search result should reflect an actual match within the indexed content or metadata.

The system should not manufacture relationships between words that merely sound similar or belong to broadly related subjects.

2.5 Transparent Results

Readers should be able to understand why an article appeared in the results.

Where practical, the matching field or a relevant excerpt should be displayed.

2.6 Privacy-Conscious Operation

The initial implementation should preferably use a locally downloaded static search index and browser-side searching.

The search feature should not require a visitor account or transmit ordinary search queries to an external search service.

2.7 Progressive Development

Begin with reliable exact and partial text matching.

More advanced linguistic processing should be added only when it can be implemented and tested accurately for the relevant language.

3. Search Scope

The search system should cover the publicly published content intended for discovery.

3.1 Individual Literary Entries

The entries collection should form the principal search corpus.

Searchable article information may include:

Only fields that actually exist in a published entry or are generated reliably from its content should be indexed.

3.2 Weekly Collections

The weeks collection may also be indexed for searches concerning:

A weekly collection result should be distinguishable from an individual literary article.

3.3 Informational Pages

Selected public informational pages may be included when they improve discovery.

Examples include:

Technical documentation should not automatically dominate ordinary literary search results.

A future search implementation may provide a separate documentation-search option if there is sufficient demand.

3.4 Excluded Content

The search index should exclude:

The index should contain only material intended for public discovery.

4. Search Entry Point

A dedicated search page may be introduced at:

/search/

This URL is a proposed route and should not be treated as an implemented page until it has been created and tested.

4.1 Search Interface

The search page should provide:

The initial interface should remain simple.

Avoid displaying many filters before there is sufficient archive content to make them useful.

4.2 Search Field

The search input should accept:

The input should support ordinary copy-and-paste operations and correctly preserve Unicode text.

4.3 Search Activation

Search should support:

Do not require a mouse or pointer interaction.

5. Multilingual Search Requirements

Multilingual support is the most important distinguishing feature of this search system.

Different writing systems have different character properties, spelling conventions, and linguistic structures.

A single universal normalisation rule cannot safely handle every language.

5.1 Unicode Support

The search system should support Unicode text.

This includes the writing systems used for:

The implementation should preserve original text without converting the stored literary expression into a simplified representation.

5.2 Unicode Normalisation

Where appropriate, search matching should use Unicode normalisation, with NFC as a suitable initial baseline.

Normalisation should be applied to a separate search representation rather than replacing the original stored text.

This distinction is important because:

Do not apply destructive transformations to the original text merely to improve search convenience.

5.3 Original-Script Matching

A query in the original script should be able to find matching passages in the original literary text.

For example:

An Urdu query should be compared with the Urdu text field rather than requiring the visitor to know its English transliteration.

Likewise, a Tamil query should be able to find Tamil material directly.

5.4 Transliteration Matching

Where an article has a transliteration field, that field should be independently searchable.

A Latin-script query may therefore find an article whose original text is written in another script.

However, transliteration is not a universal one-to-one representation.

Different writers may use different Roman spellings for the same original expression.

Search should accommodate documented alternatives where useful without silently treating every possible spelling as identical.

5.5 English Translation Matching

Where an English translation is published, its text should also be indexed.

An English query may therefore discover an article through its translated expression.

Search results should identify that the match occurred in a translation rather than implying that the English phrase appears in the original work.

5.6 Language-Specific Matching

Language-specific matching rules may eventually be introduced for:

These rules should be implemented only after testing with real material from the relevant language.

5.7 Urdu Bidirectional Text

Search should support right-to-left Urdu input and mixed Urdu-English queries.

The interface should:

Search input direction may be determined using suitable bidirectional handling rather than forcing every query to use the same direction.

6. Searchable Metadata

Search should include relevant metadata alongside article text.

6.1 Article Identity

Useful searchable fields include:

6.2 Creator and Attribution Fields

Where present and verified:

These fields should retain their distinct roles.

A search for a singer should not imply that the singer wrote the original lyrics.

6.3 Language Metadata

Include:

This allows a reader to search for a language by either its English or original-script name where the search index contains both.

The actual language identity must continue to come from _data/languages.yml and the article’s corresponding metadata.

6.4 Categories and Tags

Where implemented and consistently maintained, search may index:

The search system should use the controlled vocabulary established in CATEGORIES_AND_TAGS.md.

6.5 Commentary

The article’s principal English commentary may be indexed.

This allows readers to discover articles through ideas discussed in the commentary even when those ideas do not appear in the original literary expression.

A match in commentary should be distinguishable from a match in the original text or translation where practical.

7. Proposed Search Index

A lightweight generated search index is the preferred initial technical approach.

A possible generated resource is:

/search-index.json

This is a proposed output file, not a file that currently exists merely because it is documented here.

The search index should contain structured information derived from published pages.

An individual article record may contain:

Not every record requires every field.

Optional fields should be omitted or represented consistently when unavailable.

The index should avoid repeating information unnecessarily.

7.2 Example Record

The following illustrates the conceptual structure of one search record.

{
  "id": "example-literary-exploration",
  "type": "entry",
  "title": "A Literary Exploration",
  "url": "/verse-variable/entries/example-literary-exploration/",
  "description": "An exploration of literary expression and meaning.",
  "date": "2026-11-12",
  "week_id": "week-2026-46",
  "language_id": "urdu",
  "language_code": "ur",
  "language_name": "Urdu",
  "author": "Example Author",
  "original_text": "Original literary text",
  "transliteration": "Original literary text in Latin script",
  "translation": "English translation",
  "categories": ["poetry"],
  "tags": ["form:ghazal", "theme:memory"],
  "content": "Published article commentary"
}

This is a schema illustration, not actual article content.

7.3 Original URL Handling

Search results should link to the established canonical article URL.

The implementation must account for the project’s base URL:

/verse-variable

Avoid generating broken URLs by confusing the repository’s GitHub Pages project path with the domain root.

7.4 Index Generation

The search index should preferably be generated at build time from the actual published Jekyll collections.

This reduces the risk of maintaining a separate, manually edited copy of article metadata.

Where practical, the index should be generated automatically from the same source content used to build the website.

8. Search Matching and Ranking

Search results should be ordered according to relevance rather than merely publication date.

The ranking model should remain understandable and maintainable.

8.1 Initial Matching Priorities

A proposed relevance order is:

  1. Exact article title.
  2. Exact original literary text or work title.
  3. Exact author or creator name.
  4. Exact transliteration.
  5. Exact English translation.
  6. Partial title and metadata matches.
  7. Original-text matches.
  8. Translation and transliteration matches.
  9. Main commentary matches.
  10. Relevant category and tag matches.

The order may be refined after testing.

8.2 Phrase Matching

An exact phrase match should generally receive a higher relevance score than scattered individual word matches.

For example, a query containing several words in a specific order should prefer articles containing that phrase.

8.3 Partial Matching

Partial matching may help visitors who:

However, partial matching should be conservative.

Excessively broad matching can produce large numbers of irrelevant results.

8.4 Field Weighting

Different fields should receive different relevance weights.

Suggested initial priorities:

Field Relative importance
Article title Very high
Original work title Very high
Author or creator Very high
Exact original passage Very high
Transliteration High
Translation High
Main commentary Medium
Description Medium
Categories and tags Medium
General metadata Low to medium

These are design priorities rather than final numerical ranking weights.

8.5 Publication Date

Publication date may be used as a secondary ordering factor when two results have comparable relevance.

A newly published article should not automatically outrank an exact match from an older article.

Relevance should remain the principal sorting criterion.

8.6 Duplicate Results

The same article should not appear multiple times merely because the query matches several fields.

Instead, one result should display the article once, with the most useful matching excerpt or information.

9. Search Result Presentation

Every result should be concise and informative.

An individual result may display:

9.2 Match Context

Where practical, results should display a short excerpt showing why the article matched.

For example:

This distinction is particularly useful for multilingual literary searches.

9.3 Search Highlighting

Matching terms may be highlighted in the result excerpt.

Highlighting should:

Search highlighting must not introduce HTML or character corruption into original passages.

9.4 Result Types

Where multiple content types are indexed, identify them clearly.

Potential labels include:

Do not mix technical documentation and literary content without providing an understandable distinction.

10. Search Filters

Filters should be introduced progressively.

The initial search experience should not overwhelm visitors with unnecessary controls.

10.1 Language Filter

Potential language choices:

The filter should use the article’s original language metadata.

10.2 Content-Type Filter

Possible options:

10.3 Literary-Form Filter

Possible options:

10.4 Theme Filter

Possible options:

10.5 Scientific Subject Filter

Possible options:

This filter should include only articles with substantive scientific engagement.

10.6 Filter Behaviour

Filters should operate consistently with the active search query.

Clearing a filter should not unexpectedly erase the query itself.

The interface should make the selected filtering conditions visible.

11. Search Query Behaviour

The initial search implementation should support ordinary user expectations.

11.1 Empty Queries

An empty query should not generate an unnecessarily large list of results.

Instead, display a useful instruction explaining how to search the archive.

11.2 Exact Phrase Queries

Quoted phrase searching may be supported when the selected search engine can implement it reliably.

If exact phrase syntax is not implemented, do not pretend that quotation marks have special meaning.

11.3 Multiple Words

The search engine should define how it handles queries containing multiple words.

A reasonable initial approach is to prioritise records containing all meaningful query terms while allowing partial matches with lower ranking where appropriate.

The precise behaviour should be documented in the search interface.

11.4 Case Handling

For languages with case distinctions, matching may use appropriate case-folding.

The original displayed content must remain unchanged.

11.5 Spelling Variants

Where reliable alternative spellings exist for a work or person, they may be indexed as supplementary search aliases.

Such aliases should not replace the preferred display name.

11.6 Search Corrections

Automatic spelling correction or fuzzy matching should not be introduced until its behaviour has been tested with all supported writing systems.

A correction suitable for English may be inappropriate for Urdu or another language.

12. Search Interface States

The search interface should handle the following states clearly.

12.1 Initial State

Explain that visitors can search by:

12.2 Loading State

Where the search index must be loaded, communicate the loading state without unnecessarily delaying interaction.

12.3 Results State

Display the results and the number of matching records where reliably available.

12.4 No Results State

Provide a helpful message.

Suggested wording:

“No matching entries were found. Try another spelling, a shorter phrase, or a different language representation.”

Do not imply that the archive contains a work when no verified matching entry exists.

12.5 Error State

If the search index cannot be loaded, explain that search is temporarily unavailable.

Where practical, provide a link back to the main archive.

Do not display an empty results page as though the search completed successfully.

12.6 Filtered State

Clearly communicate which filters are active and provide a way to remove them.

13. Accessibility Requirements

Search must support readers with different accessibility requirements.

The implementation should aim to support WCAG 2.2 Level AA.

13.1 Input Accessibility

13.2 Result Accessibility

13.3 Multilingual Accessibility

13.4 Keyboard Behaviour

Visitors should be able to:

No essential operation should require a mouse.

14. Privacy and Data Handling

The preferred initial search architecture is local static search.

Under this design, the search index is downloaded from the publication’s own hosting location and queries are processed in the visitor’s browser.

14.1 No Account Requirement

Search should not require visitor registration or a personal account.

14.2 No External Query Transmission by Default

The initial implementation should not send ordinary search queries to third-party search providers or advertising services.

14.3 No Search Profiling

The publication should not intentionally build visitor profiles from search terms or browsing behaviour.

14.4 Browser Storage

Search history should not be stored persistently unless a clearly justified feature is introduced and its behaviour is documented.

Any optional browser-side preference should be distinguished from server-side visitor identification.

14.5 Hosting Infrastructure

Access to the search page and static search-index resource remains subject to the hosting provider’s normal technical processing and privacy practices.

Refer to PRIVACY.md for the broader privacy policy.

14.6 Future External Search Services

If an external search service is considered in the future, evaluate:

Do not introduce external query processing without reviewing its implications and updating the privacy documentation where necessary.

15. Technical Implementation

15.1 Preferred Initial Architecture

The preferred implementation is:

  1. Generate a static search index during the Jekyll build.
  2. Make the index available as a public static resource.
  3. Load the index in the browser when search is used.
  4. Process queries locally.
  5. Display matching article records.
  6. Link each result to its permanent page.

This approach is compatible in principle with a static publishing architecture and avoids requiring a dedicated search server.

Actual implementation should be validated against the project’s GitHub Pages build environment.

15.2 Jekyll Integration

The search index should be generated from the content collections rather than manually maintained.

Potential inputs include:

The generated output must reflect the actual published content.

15.3 JavaScript

Client-side JavaScript may be used for:

The feature should fail gracefully if JavaScript or the search index is unavailable.

The main archive should remain available as an alternative discovery route.

15.4 Search Engine Selection

The initial implementation should prefer a lightweight solution.

Potential approaches include:

No specific library is mandatory.

Before selecting one, evaluate:

15.5 GitHub Pages Compatibility

The current project uses Jekyll with a lightweight configuration.

Do not introduce server-side search infrastructure or unsupported build plugins without reviewing the hosting arrangement.

Any additional dependency should be documented in the repository and its applicable licence recorded in ASSET_CREDITS.md.

15.6 Index Size and Performance

As the archive expands, search performance should be monitored.

Possible improvements include:

Do not remove important original-language text solely to reduce index size without considering the consequences for multilingual discovery.

16. Security and Input Handling

Search queries and indexed content should be handled safely.

The implementation should:

Matching and highlighting should operate on safe text representations.

The search system must not interpret an ordinary search query as executable code.

17. Search Quality Testing

Testing should use actual material from the publication’s supported languages.

17.1 English

Test:

17.2 Urdu

Test:

17.3 Hindi and Marathi

Test:

17.4 Kannada, Tamil, Malayalam, and Telugu

Test:

17.5 Metadata

Test:

17.6 Error Handling

Test:

17.7 Relevance

Review whether:

18. Search Acceptance Criteria

The initial search feature should not be considered ready until it satisfies the following conditions.

Functional

Multilingual

Accessibility

Privacy and Security

Technical

19. Progressive Implementation Plan

Search should be introduced in manageable stages.

Implement:

Add:

Stage 3 — Search Filters

Introduce:

Use only filters supported by sufficiently complete metadata.

Stage 4 — Advanced Discovery

Consider:

Advanced features should be introduced only after they have been tested with representative literary material.

20. Relationship with Other Documentation

This specification should be maintained alongside:

Search should use the same language identifiers, metadata conventions, classification vocabulary, and accessibility principles established in these documents.

21. Future Development

The long-term objective is to make the Verse & Variable archive searchable across linguistic and intellectual boundaries.

Future capabilities may include:

These capabilities should emerge from the actual content of the archive and from demonstrated reader needs.

The aim is not to create the most technically elaborate search engine. It is to create a dependable discovery system that helps readers encounter literature, ideas, and scientific connections they might otherwise never find.


Verse & Variable
A line of verse. A line of inquiry.
لفظ سے لہر تک — Lafz se Lehar tak