Indexes
Yes. Because your issueProgress and issueType enums are stored as flat string fields (and not inside arrays), you can use Ascending/Descending indexing in a production environment. You do not need arrayConfig (which is strictly for ARRAY_CONTAINS indexing). [1, 2, 3] However, since you are mixing an in query with multiple fields, there are exactly three production gotchas you must account for to avoid query errors or scaling performance bottlenecks:
THIS IS ONLY A LIMITATION ON FIRESTORE STANDARD
Firestore converts in and or queries into a list of separate query combinations (disjunctions) under the hood. [5]
- The Limit: The maximum total number of combinations allowed in a single query is 30.
- How it affects you: Since you are querying issueProgress for [‘open’, ‘waiting’], that costs 2 disjunctions. If you ever change the query to search for multiple issueType options simultaneously using an or block later, you must ensure (number of progress states) × (number of types) <= 30. [5]
If you add an orderBy() clause to sort your results (e.g., sorting issues by createdAt), the field order inside your Firestore Composite Index matters significantly.
- The Rule: Fields filtered with equality (==) or in should always come before fields used for inequality (>, <) or sorting (orderBy) inside the index definition. [6, 7]
- Correct Index Configuration:
- userId (Ascending) — Equality 2. issueType (Ascending) — Equality 3. issueProgress (Ascending) — The “in” filter 4. createdAt (Ascending or Descending) — Your sorting field
If you only have single-value fields, you will not hit exploding indexes. However, ensure that no other field in this specific collection uses arrayConfig or contains an array if it’s tied to this composite index. Including an array field inside a multi-field composite index multiplies the index entry count exponentially, eating up your Firestore index write quotas. [8]
- Use in for your query.
- Keep issueProgress and issueType configured as Ascending or Descending fields in your composite index (do not use array options).
- Ensure your in array length stays under 30 elements.
- Put your sorting fields last in the composite index layout. [1, 2, 5, 6]
Since you are using Firestore Enterprise Edition with Pipeline operations (evident from your use of .pipeline(), Field.of(), regex_match, and equal_any), the traditional limits do not apply to you in the same way. [1, 2] Firestore Enterprise’s advanced query engine completely removes the 30-disjunction limit and handles cross-multiplied queries seamlessly without index constraints. [2] However, because your query relies heavily on dynamic filtering and pagination (offset and limit), you have a silent logic bug waiting to happen if you cross your enums. Here is the code adjustment to prevent it, followed by index clarifications.
Your current implementation feeds all your filters (including both equal_any statements) straight into a single database pipeline:
pipeline = pipeline.where(And(*filters))
If a user checks 5 progress elements and 6 type elements, Firestore Enterprise will execute it successfully. However, if your collection grows into millions of records, combining multiple heavy equal_any arrays alongside a runtime regex match (regex_match) can dramatically slow down pipeline execution. [2, 3, 4] Furthermore, because you use .offset() for pagination, Firestore still has to scan and skip matching documents under the hood. If you need to switch to an application-side fallback strategy for ultra-fast performance, you cannot safely mix pagination inside the database. [4]
Because you have Enterprise, keep the heavy lifting on the database, but safely split filtering or optimize the schema setup. If you ever find your query latency increasing, use this pattern to drop the lighter enum filtering down to Python memory right before validation:
# --- Build Filters ---
filters = []
if request.query:
filters.append(Field.of("message").regex_match(f"(?i){request.query}"))
# KEEP the larger array on the DB Engine
if request.filter_by_issue_type:
filters.append(
Field.of("issue_type").equal_any([t.value for t in request.filter_by_issue_type])
)
# ENTERPRISE WORKAROUND: If both enums are highly populated,
# extract progress filtering out of the DB pipeline and handle it in Python memory.
# (Only do this if you notice performance drops, otherwise Enterprise handles equal_any natively)
db_handles_progress = True
if request.filter_by_issue_progress and db_handles_progress:
filters.append(
Field.of("progress").equal_any([p.value for p in request.filter_by_issue_progress])
)
No. Even inside Firestore Enterprise, arrayConfig is strictly reserved for variables saved as list structures in your NoSQL document (e.g., tags = [“bug”, “urgent”]). Because your issue_type and progress fields are stored as single strings per document, your index configuration must remain Ascending or Descending.
- Indexes are Optional: For Firestore Enterprise pipelines, index creation is completely optional. The pipeline engine can execute your equal_any and regex_match filters dynamically without a pre-defined composite index. [2]
- When to define one: If your collection scales past hundreds of thousands of documents, you should explicitly force a composite index to maintain O(1) lookup speeds. [5]
- The Layout: If you build a manual index for this pipeline, organize it from highest cardinality (most unique values) to lowest:
- from_uid / to_uid (Ascending)
- issue_type (Ascending)
- progress (Ascending)
- Your sorting field (e.g., created_at Descending) [6]
If you want, tell me:
- What is the average volume of documents matching an individual user (owner_uid)?
- Are you noticing any latency delays when executing the regex_match filter on the message field?
I can help you convert the cursor structure from .offset() to .start_after() to make your pagination drastically faster.
In a Firestore architecture using a Flutter frontend and a Python backend (Cloud Functions), start_after is a cursor-based pagination method. [1, 2] Instead of telling the database “skip 50 rows” (which is what .offset(50) does), a cursor tells the database “find the exact document I point to, and start reading the very next record after it”. [1, 3, 4]
Using .offset((page_num - 1) * page_size) seems easy, but it has two major production penalties in Firestore: [3, 5]
- You are over-billed: If you request Page 10 (skipping 450 items to read 50), Firestore physically reads all 500 documents under the hood. You get billed for 500 document reads, not 50. [3, 4]
- Performance drops: As users flip to deep pages, queries become slower because Firestore has to repeatedly scan and skip past previous pages. [4, 6, 7, 8]
Because your Python Callable Backend cannot see the raw Dart/Flutter memory, you cannot pass a live “Document Snapshot” directly. Instead, you pass the Last Document ID and the values of your sorting fields from Flutter to Python. [9, 10, 11]
Modify your Request schema adapter to expect a pagination cursor object instead of a raw page_num.
Add these tracking fields to your incoming Request Schema adapterclass GeneralIssueSearchRequestAdapter(BaseModel):
# ... your existing filters ...
page_size: int = 20
# Cursor parameters
last_doc_id: str | None = None
last_sort_value: Any | None = None # The value of the primary sorting field (e.g., timestamp or count)
Modify your .offset() logic to use .start_after(). In Firestore Enterprise Pipelines, you provide an array of values matching the fields you sorted on, plus the unique document ID to break ties. [12, 13, 14]
# --- Sorting ---
sort_type = request.sort_type
for field, direction in sort_type.pipeline_sort_fields:
pipeline = pipeline.sort(
Field.of(field).ascending() if direction == "asc" else Field.of(field).descending()
)
# --- Efficient Pagination (Replaces Offset) ---
page_size = request.page_size or MAX_ENTERPRISE_PAGE_SIZE
if request.last_doc_id and request.last_sort_value:
# Tell the pipeline engine exactly where the last page ended
pipeline = pipeline.start_after([request.last_sort_value, request.last_doc_id])
pipeline = pipeline.limit(page_size)
Your Flutter application needs to know the specific values of the last record so it can request the next page. Modify your schema to include these markers: [4, 15]
# Capture the last document's data to return as cursor references
next_last_doc_id = docs[-1].id if docs else None
# Pull the value of whatever field you dynamically sorted on (e.g., 'created_at')
primary_sort_field = sort_type.pipeline_sort_fields[0][0] if sort_type.pipeline_sort_fields else "id"
next_last_sort_value = getattr(docs[-1], primary_sort_field, None) if docs else None
return GeneralIssueResponseSchema(
page_size=page_size,
issues=docs,
last_doc_id=next_last_doc_id,
last_sort_value=next_last_sort_value
)
On your mobile UI, your state manager or pagination scrolling listener keeps track of the returned cursor dictionary. [16]
// Track your page cursor tokens in your Flutter State/Controller
String? lastDocId;dynamic lastSortValue;
List
Future
final response = await callable.call(<String, dynamic>{ ‘query’: uiSearchQuery, ‘page_size’: 20, ’last_doc_id’: lastDocId, // Null on page 1, populated on subsequent scrolls ’last_sort_value’: lastSortValue, // Null on page 1, populated on subsequent scrolls });
final data = response.data;
// 1. Update the state with new items setState(() { issues.addAll(parseIssues(data[‘issues’]));
// 2. Save the cursor markers for the NEXT scroll trigger
lastDocId = data['last_doc_id'];
lastSortValue = data['last_sort_value'];
}); }
- O(1) Constant Speed: Page 1 and Page 1,000 load instantly at identical database cost.
- Real-Time Consistency: If a new document is inserted on Page 1 while a user is reading Page 3, cursor pagination ensures items don’t shift awkwardly or duplicate on screen like they do with offset pagination. [4, 17]
If you want, tell me what specific fields you allow users to sort by in your sort_type.pipeline_sort_fields logic (e.g., Datetime, Priority, ID). I can show you how to parse and sanitize those raw timestamp objects across the Python/Dart JSON layer safely.