fix: await get catalog in datafusion integration test and update version in python binding

[lliangyu-lin]

Which issue does this PR close?

• Closes #.

What changes are included in this PR?

• Previous feat(datafusion): Implement insert_into for IcebergTableProvider #1600 added few integration tests, but need to include the new changes introduced in memory catalog loader change feat(catalog): Implement catalog loader for in memory #1623 .
• This caused build failures

Are these changes tested?

Yes, existing tests

[liurenjie1024]

Thanks @lliangyu-lin for this fix!

[liurenjie1024]

Fix the comments?

[lliangyu-lin]

Hi @liurenjie1024, I think the python binding error is still not fixed yet. The build is passing though after adding the .await(), but python binding seems like a different error, which might need deeper look.

We can probably merge this to fix the build failure first

[liurenjie1024]

Hi @liurenjie1024, I think the python binding error is still not fixed yet. The build is passing though after adding the .await(), but python binding seems like a different error, which might need deeper look.

We can probably merge this to fix the build failure first

Thanks @lliangyu-lin for the effort. I don't think we should merge this without actually fixing the ci failure. I'll do some investigation to about it.

[lliangyu-lin]

@liurenjie1024 I tried revert the insert into support commit and rerun the tests locally, but the tests are still failing. I'm suspecting something to do with the datafusion and arrow versions. Pinning the version to arrow 16 and datafusion 47 seems to fix the issue for now.

[Fokko]

While I don't think it should be that picky on the Arrow version, I think it would be good to get this in to get the CI unblocked. Thanks @lliangyu-lin for working on this, and @liurenjie1024 fro the review 🙌

[manuzhang]

I think it's due to apache/datafusion-python#1217. We can loosen arrow version after @kevinjqliu getting #1639 in