Reindexer

[chaporgin]

Hi! Thank you for the library. I see that Reindexer is disabled. Can you please let me know what the plans are about the Reindexer and how the fact that Reindexer is doing nothing ATM because of IndexNames being empty by default affects the production systems?

[bgentry]

@chaporgin hi there, the reindexer was disabled before release because we felt it needed more testing before being enabled by default. Although the maintenance process is still active, the overhead is very low: on the single leader node it will run its loop daily at midnight, looping through the 0 indexes in that default list before sleeping again.

The downside of not reindexing periodically is your indexes can get quite bloated and take up significantly more space than necessary, and the bloat can cause less efficient index operations. For example I just checked on the demo app's database which hasn't been reindexed in a very long time, and discovered that the args index had grown to 4 GB:

riverdemo=# SELECT
    indexrelname AS index_name,
    pg_size_pretty(pg_relation_size(indexrelid)) AS index_size
FROM pg_stat_user_indexes;
               index_name               | index_size 
----------------------------------------+------------
 river_job_prioritized_fetching_index   | 80 MB
 river_job_args_index                   | 4013 MB
 river_job_metadata_index               | 645 MB
...

After reindexing these 3 rows, the size was much smaller:

riverdemo=# SELECT                                    
    indexrelname AS index_name,
    pg_size_pretty(pg_relation_size(indexrelid)) AS index_size
FROM pg_stat_user_indexes;
               index_name               | index_size 
----------------------------------------+------------
 river_job_prioritized_fetching_index   | 38 MB
 river_job_args_index                   | 35 MB
 river_job_metadata_index               | 5656 kB
...

It's probably past time for us to take a serious look at turning this functionality on by default and seeing if we can address the potential downsides that led us to disable it before initial release. cc @brandur

[bgentry]

The demo app does a little under a million jobs a day (I believe it has the default 24 hour retention). Multiply that by many months and that’s a lot of jobs!

The reindexer uses REINDEX INDEX CONCURRENTLY which is available in newer Postgres versions so it does not block writes. I believe there was some concern about needing more knobs for controlling the timeout for these queries, and also wanting to make sure we don’t have multiple reindexing attempts going on the same index at one time. They’re solvable problems that we just ran out of time to sort out before release and haven’t yet come back around to.

[bgentry]

@chaporgin the change to enable the reindexer by default was just merged in #718 and will go out in the next release. It runs on the leader daily at midnight UTC by default, but can be overridden in the client config. It also only reindexes the GIST indexes that seem to have consistent growth, whereas the b-tree indexes stay fairly constrained.

[chaporgin]

Thank you for that change!

I have been reading the code, and I'm wondering what purpose the 2 GIN-based indexes serve. I am talking about the

CREATE INDEX river_job_args_index ON public.river_job USING gin (args);
CREATE INDEX river_job_metadata_index ON public.river_job USING gin (metadata);

It does not look like JobListParams allows to fetch by the args. Or does it?

river/job_list_params.go

Lines 167 to 177 in bf286e4

	type JobListParams struct {
	after *JobListCursor
	kinds []string
	metadataFragment string
	overrodeState bool
	paginationCount int32
	queues []string
	sortField JobListOrderByField
	sortOrder SortOrder
	states []rivertype.JobState
	}

I see why the second index, on the metadata field, might be helpful - because it's possible to set metadata in Metadata method. However, I do see usage of that only in tests.

Correct me if I'm wrong, but I don't see the usage of both of them ATMs.

[bgentry]

Those indices primarily exist to support UI features around searching by args and metadata, though that's not fully implemented yet. They're also used by pro features.

You're correct that the built-in job list query doesn't currently have an option for arg filtering.

[chaporgin]

works as expected, thank you!