Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rr/sc 60366 sparse global order reader merge #5417

Open
wants to merge 167 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
167 commits
Select commit Hold shift + click to select a range
f3ea201
Add cpp parallel merge implementation
rroelke Dec 4, 2024
ba90274
Move template parallel merge impl into header and template Compare fu…
rroelke Dec 4, 2024
b90353b
Domain::dimensions
rroelke Dec 4, 2024
54a7de2
Remove trailing newline
rroelke Dec 5, 2024
ef488f8
TransientReadState::contains_tile
rroelke Dec 5, 2024
900ebb2
void SparseGlobalOrderReader<BitmapType>::compute_result_tile_order()…
rroelke Dec 5, 2024
334eb0d
Fix uint64_t in ParallelMerge
rroelke Dec 5, 2024
399023b
Fix errors in compute_result_tile_order
rroelke Dec 5, 2024
0593e30
Fix indexing error in GlobalOrderMBRCmp
rroelke Dec 5, 2024
007e43d
Change test section names so that -c option can be used
rroelke Dec 5, 2024
169b20e
Passing 'qc removes full tile' unit test with compute_result_tile_order
rroelke Dec 5, 2024
a8cd4b0
Remove per-fragment memory fields
rroelke Dec 5, 2024
a9d78d0
TEST_CASE parallel merge rapidcheck verify_split_point_stream_bounds
rroelke Dec 6, 2024
0299ca7
ParallelMergeException
rroelke Dec 6, 2024
804fc6f
VerifyIdentifyMergeUnit
rroelke Dec 6, 2024
0285add
VerifyTournamentMerge
rroelke Dec 6, 2024
54cbed8
VerifyParallelMerge
rroelke Dec 6, 2024
0183686
Add more parallel merge test cases for uint8_t and numbers in range 0,10
rroelke Dec 6, 2024
da5d191
Remove commented-out code
rroelke Dec 6, 2024
1225c21
Fix exceeding memory budget
rroelke Dec 9, 2024
3833eaa
Add UntypedDatum methods to Range
rroelke Dec 9, 2024
5a20d6b
Refactor GlobalCmp
rroelke Dec 9, 2024
3c6e4b2
Add fragment skew, fragment interleave tests
rroelke Dec 9, 2024
d281adb
Compare against next tile MBR in add_next_cell_to_queue
rroelke Dec 9, 2024
b43381d
Fill in memory budget exception
rroelke Dec 10, 2024
d2d06fa
Add TODO for test case
rroelke Dec 10, 2024
a55ddf3
tdb::pmr::emplace_unique
rroelke Dec 10, 2024
fc8936e
Use pmr in parallel merge
rroelke Dec 10, 2024
a2c6a8d
Test tweaks
rroelke Dec 10, 2024
9023c1c
Relax condition for throwing 'Cannot load a single tile'
rroelke Dec 10, 2024
83527f0
emplace_unique is not a violation
rroelke Dec 10, 2024
d335802
Consolidate and refactor comparators
rroelke Dec 10, 2024
c7e2ebb
Avoid repeat constant in fragment skew test
rroelke Dec 10, 2024
477e72a
CSparseGlobalOrderFx::run_1d, used for fragment skew test
rroelke Dec 11, 2024
89f401c
Add rapidcheck for fragment skew test
rroelke Dec 11, 2024
a3f22c9
Add rapidcheck for fragment interleave test
rroelke Dec 11, 2024
40e2ad6
Re-name new fields
rroelke Dec 11, 2024
5435a87
Fix budget for fragment skew test
rroelke Dec 11, 2024
9b84911
Add config switch to run old code by default
rroelke Dec 11, 2024
9566577
Fix config unit test
rroelke Dec 11, 2024
6d2797e
Also don't do preprocess mode for hilbert
rroelke Dec 12, 2024
d946d29
RAII close array in tests and macro to multiplex catch/rapidcheck ass…
rroelke Dec 12, 2024
39d6268
Fix slab length
rroelke Dec 12, 2024
333b27d
DefaultArray1DConfig
rroelke Dec 12, 2024
8611a34
Arbitrary extent in fragment skew test
rroelke Dec 12, 2024
1978db0
Arbitrary<FxRun1D>
rroelke Dec 12, 2024
02bb56e
Fix queue-to-bound comparison
rroelke Dec 12, 2024
d4c7e7f
Use CApiArray to RAII close in write_1d_fragment
rroelke Dec 12, 2024
b70634a
rapidcheck sparse global order test for arbitrary 1D fragment set
rroelke Dec 12, 2024
bfe5de8
Clean CMakeLists.txt
rroelke Dec 13, 2024
25f3f01
Fix allow_dups in fragment skew, interleave tests
rroelke Dec 13, 2024
3d491be
Merge remote-tracking branch 'origin/dev' into rr/sc-60366-sparse-glo…
rroelke Dec 16, 2024
f5bde2d
parallel merge throws if status is not ok
rroelke Dec 16, 2024
5b44b63
LIBRAPIDCHECK REQUIRED
rroelke Dec 17, 2024
e486fcd
Add harness for running comparison off/on against any array URI
rroelke Dec 17, 2024
263c23f
Add subarray to fragment skew test
rroelke Dec 17, 2024
e47fc5b
Remove dead SECTION
rroelke Dec 18, 2024
e39bbbc
Add signal test for supporting multi-range subarray for global order
rroelke Dec 18, 2024
b3c136a
Improve populate_relevant_tiles
rroelke Dec 18, 2024
e13e65c
make format
rroelke Dec 18, 2024
7d9443a
TimeKeeper::dump_durations
rroelke Dec 18, 2024
3460322
Genericize rapidcheck generators for domain,coord,fragment,etc
rroelke Dec 18, 2024
52ecdb3
Add test for merge being too wide
rroelke Dec 18, 2024
ecef2ca
ParallelMergeFuture destructor, and some other methods
rroelke Dec 19, 2024
d12ba41
Do not block on whole merge to begin creating result tiles
rroelke Dec 19, 2024
4555c40
Add test for many overlapping fragments being too wide for memory budget
rroelke Dec 19, 2024
73fb629
Call can_complete_in_memory_budget in run_1d
rroelke Dec 19, 2024
1c495b7
Remove commented-out code
rroelke Dec 20, 2024
b555593
show<Domain<int>> specialization
rroelke Dec 20, 2024
872f416
template Range constructor
rroelke Dec 20, 2024
82c2871
Rapidcheck for many overlapping fragment test
rroelke Dec 20, 2024
3a40589
Add subarrays to existing tests
rroelke Dec 20, 2024
3e93a51
datatype_traits
rroelke Dec 23, 2024
5b9ed6d
Genericize write_fragment and create_array
rroelke Dec 23, 2024
383991d
Generic expectcoords, expectatts
rroelke Dec 24, 2024
40cd0a8
Fix types to auto
rroelke Dec 24, 2024
f61f7c5
Generic sort for naive comparison
rroelke Dec 24, 2024
b04b5e0
Genericize query loop
rroelke Dec 24, 2024
8091634
Genericize expectatts check
rroelke Dec 29, 2024
04aab6c
run_1d => run
rroelke Dec 29, 2024
53cfd7d
Genericize can_complete_in_memory_budget
rroelke Dec 30, 2024
d58c92a
FxRun2D instantiation compiles
rroelke Dec 30, 2024
dab89d0
Fixed release build unused variable
rroelke Dec 30, 2024
b5bc99c
::run exception for 'Cannot load tile offsets'
rroelke Dec 30, 2024
8db905b
Apply subarray to rapidcheck 2d
rroelke Dec 30, 2024
c539acd
show<FxRun2D>
rroelke Dec 30, 2024
0c92556
Fix CSparseGlobalOrderFx::run to sort 'expect' in global order
rroelke Dec 30, 2024
010f83b
vfs_test_get_fs_vec requires S3 to add rest-s3
rroelke Dec 31, 2024
c793f47
Use VFS in CSparseGlobalOrderFx and add error detail to helpers.cc RE…
rroelke Dec 31, 2024
29441ec
Update unit test for REST
rroelke Dec 31, 2024
e44c707
Serialization of preprocess works
rroelke Dec 31, 2024
628e7d6
Change config parameter to be uint64_t determining merge unit min size
rroelke Dec 31, 2024
76325d4
Mkae FsLocal the default
rroelke Dec 31, 2024
beb9604
Formatting and comments
rroelke Dec 31, 2024
12b0b1e
Tile order accounting against memory budget
rroelke Jan 2, 2025
782900a
Do not shrink by default for rapidcheck 1D, 2D tests
rroelke Jan 2, 2025
0012cb8
tdb::make_shared for ResultTileCmp
rroelke Jan 2, 2025
34a9a74
Self review cleaning: remove FxFragment1D, tuple.h
rroelke Jan 2, 2025
8441a20
Fix librapidcheck linking
rroelke Jan 2, 2025
e83713c
comments in unit-sparse-global-order-reader.cc
rroelke Jan 2, 2025
109b223
std::integral T for Range constructor
rroelke Jan 3, 2025
99dbbfc
Hidden tag for benchmark test
rroelke Jan 3, 2025
2f5199f
Merge remote-tracking branch 'origin/dev' into rr/sc-60366-sparse-glo…
rroelke Jan 3, 2025
1cd4169
Turn on by default
rroelke Jan 3, 2025
d9a93fc
Use default value directly in unit-capi-config.cc
rroelke Jan 3, 2025
781b876
Fix erroneous recomputation of tile order after completion
rroelke Jan 3, 2025
76d80a3
Move tuple.h to stdx directory
rroelke Jan 3, 2025
d3253d8
Split up rapidcheck/array.h file and change namespace
rroelke Jan 3, 2025
de37245
Comment explains parallel merge algorithm, and simplify split_point_s…
rroelke Jan 3, 2025
efdeb53
Move CApiArray to array_helpers.h
rroelke Jan 3, 2025
46f67d9
Refactor SparseGlobalOrderMemoryBudget
rroelke Jan 3, 2025
ab56a4d
preprocess_tile_order_.num_tiles_ = 0
rroelke Jan 3, 2025
ee32782
Move query_applicator into array_templates.h and change macro to ASSE…
rroelke Jan 3, 2025
7372f6f
Use function for query_applicator to determine field name
rroelke Jan 3, 2025
af81ad6
has_more_tiles() checks enabled_
rroelke Jan 6, 2025
8279a05
Add AsserterRuntimeException
rroelke Jan 6, 2025
f42e3a4
query_applicator::set_field uses 0-based lambda arg
rroelke Jan 6, 2025
d5f005e
tiledb_submit_a_b
rroelke Jan 6, 2025
6b8a166
tiledb_submit_a_b load config from json
rroelke Jan 6, 2025
022eb0b
Fix num_tiles in can_complete_in_memory_budget
rroelke Jan 6, 2025
d4d86ec
Fix make_extent
rroelke Jan 6, 2025
f3b7340
Remove commented-out code
rroelke Jan 6, 2025
3437bf7
Fix formatting
rroelke Jan 6, 2025
80694e3
tiledb_submit_a_b subarray scratch
rroelke Jan 6, 2025
b80fe81
make format
rroelke Jan 6, 2025
2787ba2
override in SparseGlobalOrderReader decls
rroelke Jan 6, 2025
a808c54
#include for std::clamp
rroelke Jan 6, 2025
fc42e28
Fix unit-sparse-global-order-reader.cc warnings hopefully
rroelke Jan 6, 2025
3c4fc5d
Fix non-gcc compiler issues
rroelke Jan 6, 2025
0d767bd
Use concept to make parallel merge much more interface friendly
rroelke Jan 6, 2025
7d857b0
Add TEST_CASE with identify split point example
rroelke Jan 7, 2025
585707d
Next round of msvc,etc compiler errors
rroelke Jan 7, 2025
3874d70
Fix MSVC errors
rroelke Jan 7, 2025
50e81c2
Non-gcc compiler fixes take 15
rroelke Jan 7, 2025
aec3e05
Some sanitization for MSVC
rroelke Jan 7, 2025
84743ac
Cmpable => Comparable
rroelke Jan 7, 2025
d823062
'&& requires' instead of 'and requires'
rroelke Jan 7, 2025
8ea9d7a
Improve rapidcheck in cmake
rroelke Jan 7, 2025
7e79b1e
Multi-range subarray signal test is more explicit
rroelke Jan 8, 2025
c49d4a0
Merge remote-tracking branch 'origin/main' into rr/sc-60366-sparse-gl…
rroelke Jan 8, 2025
8080705
Revert fs order change to vfs_helpers.cc
rroelke Jan 9, 2025
9641e3b
Print error message if tiledb_object_type fails
rroelke Jan 9, 2025
88b11c9
Add docs to tiledb_submit_a_b.cc and make results order configurable
rroelke Jan 9, 2025
6ceb242
Fix JSON in example
rroelke Jan 9, 2025
95f455f
Log exceptions to stderr
rroelke Jan 9, 2025
1a6f45f
Return status from do_submit
rroelke Jan 9, 2025
61dc130
tiledb_submit_a_b assert that results are in global order
rroelke Jan 11, 2025
692e517
Add 2D out-of-order MBR test
rroelke Jan 13, 2025
92e70d7
UntypedDatumView(std::string_view)
rroelke Jan 13, 2025
b67a0a1
Bury some gory details in array_templates.h
rroelke Jan 14, 2025
2bed928
tiledb_submit_a_b asserts that produced tuples are in global order
rroelke Jan 15, 2025
2efe0fa
Preprocess merge bound is not the next MBR, fixes performance issue f…
rroelke Jan 15, 2025
e368b9f
Acknowledge unordered in tiledb_submit_a_b
rroelke Jan 15, 2025
b756351
Review comment docstrings
rroelke Jan 15, 2025
620c941
auto apirc
rroelke Jan 15, 2025
91148f7
Doc for SparseIndexReaderBase::preprocess_tile_order
rroelke Jan 17, 2025
da6a621
all_fragment_tile_order_ => per_fragment_memory_state_
rroelke Jan 17, 2025
0495206
Doc comment for preprocess_compute_result_tile_order
rroelke Jan 17, 2025
f2ad89b
Remove maybe_ from name
rroelke Jan 17, 2025
05c1e63
PreprocessTileMergeFuture::await checks merge_ first
rroelke Jan 17, 2025
dc784da
DeleteArrayGuard, use instead of manual delete in create_array
rroelke Jan 17, 2025
3ad84ce
Fix 2d rapidcheck test
rroelke Jan 17, 2025
75ef9df
Enable col major tile/cell orders for 2d tests
rroelke Jan 17, 2025
9ea370e
CSparseGlobalOrderFx uses vfs_test_setup_ context
rroelke Jan 17, 2025
6f271d7
Format oops
rroelke Jan 17, 2025
b756484
Test fragment skew 2d merge bound
rroelke Jan 18, 2025
13d6997
Fix invalid merge bound and duplicate coord corner case
rroelke Jan 18, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -447,6 +447,7 @@ target_include_directories(local_install INTERFACE ${CMAKE_SOURCE_DIR})
enable_testing()

if(TILEDB_TESTS)
find_package(rapidcheck CONFIG REQUIRED)
# Add custom Catch2 entry point that suppresses message boxes on debug assertion
# failures on Windows CI.
find_package(Catch2 REQUIRED)
Expand Down
2 changes: 1 addition & 1 deletion scripts/find_heap_api_violations.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@
"zstd_compressor.h": ["std::unique_ptr<ZSTD_DCtx, decltype(&ZSTD_freeDCtx)> ctx_;", "std::unique_ptr<ZSTD_CCtx, decltype(&ZSTD_freeCCtx)> ctx_;"],
"posix.cc": ["std::unique_ptr<DIR, UniqueDIRDeleter>", "static std::unique_ptr<char, decltype(&free)> cwd_(getcwd(nullptr, 0), free);"],
"curl.h": ["std::unique_ptr<CURL, decltype(&curl_easy_cleanup)>"],
"pmr.h": ["std::unique_ptr", "unique_ptr<Tp> make_unique("],
"pmr.h": ["std::unique_ptr", "unique_ptr<Tp> make_unique(", "unique_ptr<Tp> emplace_unique("],
teo-tsirpanis marked this conversation as resolved.
Show resolved Hide resolved
}


Expand Down
3 changes: 1 addition & 2 deletions test/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,7 @@ target_link_libraries(tiledb_unit
Catch2::Catch2
tiledb_test_support_lib
configuration_definitions
rapidcheck
)

target_link_libraries(tiledb_unit PRIVATE $<BUILD_INTERFACE:common>)
Expand Down Expand Up @@ -373,9 +374,7 @@ endif()

# CI tests
add_subdirectory(ci)
if(WIN32)
add_subdirectory(performance)
endif()

add_custom_target(
check-package
Expand Down
25 changes: 25 additions & 0 deletions test/performance/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@
# THE SOFTWARE.
#

if(WIN32)

Comment on lines +29 to +30
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover?

# These options not exposed in bootstrap script.
option(TILEDB_TESTS_AWS_S3_CONFIG "Use an S3 config appropriate for AWS in tests" OFF)

Expand Down Expand Up @@ -94,3 +96,26 @@ if (${CMAKE_SYSTEM_NAME} MATCHES "Linux" AND CMAKE_CXX_COMPILER_ID STREQUAL "GNU
LINK_FLAGS "-Wl,--no-as-needed -ldl"
)
endif()

endif()


add_executable(
tiledb_submit_a_b EXCLUDE_FROM_ALL
$<TARGET_OBJECTS:TILEDB_CORE_OBJECTS>
"tiledb_submit_a_b.cc"
)

add_dependencies(tiledb_submit_a_b tiledb_test_support_lib)

Comment on lines +109 to +110
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
add_dependencies(tiledb_submit_a_b tiledb_test_support_lib)

target_link_libraries already adds the dependency.

target_include_directories(
tiledb_submit_a_b BEFORE PRIVATE
${TILEDB_CORE_INCLUDE_DIR}
${TILEDB_EXPORT_HEADER_DIR}
)

target_link_libraries(tiledb_submit_a_b
PUBLIC
TILEDB_CORE_OBJECTS_ILIB
tiledb_test_support_lib
)
Loading
Loading