[SPARK-42242][BUILD] Upgrade `snappy-java` to 1.1.9.1 by dongjoon-hyun · Pull Request #39811 · apache/spark

dongjoon-hyun · 2023-01-30T23:29:12Z

What changes were proposed in this pull request?

This PR aims to upgrade snappy-java to 1.1.9.1.

Why are the changes needed?

This version has the following bug fixes.

Use original compressed and uncompressed buffer's position (Use original compressed and uncompressed buffer's position xerial/snappy-java#293)
Fixed running snappy-java as OSGi bundle on Apple Silicon (M1 Pro) (#302 Fixed running snappy-java as OSGi bundle on Apple Silicon (M1 Pro) xerial/snappy-java#303)
Avoid explicit class name in throw_exception (Avoid explicit class name in throw_exception xerial/snappy-java#291)

Here is the full release note.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

dongjoon-hyun

Hi, @viirya . Could you review this? This includes your two patches.

viirya · 2023-01-30T23:34:00Z

Thanks @dongjoon-hyun . Pending CI.

dongjoon-hyun · 2023-01-30T23:34:24Z

Thank you, @viirya . Yes, pending CI!

dongjoon-hyun · 2023-01-31T00:02:53Z

Oh...

java.lang.NoSuchMethodError: java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
	at org.xerial.snappy.Snappy.compress(Snappy.java:157)

dongjoon-hyun

It seems that snappy-java is published via Java 11. I converted this PR as a draft for now.

https://github.com/xerial/snappy-java/actions/runs/4040266239/jobs/6945753237

dongjoon-hyun · 2023-01-31T00:07:26Z

Sorry, @viirya and @HyukjinKwon . I will check the situation more.

LuciferYang

+1, LGTM

LuciferYang · 2023-01-31T04:57:05Z

https://github.com/xerial/snappy-java/blob/master/build.sbt#L30

Interesting. It seems that the compilation target is still Java 8...

dongjoon-hyun · 2023-01-31T05:43:35Z

It's insufficient, @LuciferYang . The runtimeClass (rt.jar) is required during building. Otherwise, Java 11 assumes Java 11 rt.jar.

srowen · 2023-01-31T14:54:31Z

Looks like that overload isn't even in Java 11; I see it in Java 14: https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/nio/ByteBuffer.html#limit(int)
Yeah, this is not a question of which JVM you target, but of using methods not found in Java 8.

dongjoon-hyun · 2023-01-31T17:48:18Z

Ya, thank you, @srowen .

LuciferYang · 2023-01-31T18:47:49Z

https://github.com/xerial/snappy-java/releases/tag/v1.1.9.1

@dongjoon-hyun We can try to upgrade 1.1.9.1 to solve this issue

dongjoon-hyun · 2023-01-31T18:48:49Z

Thank you so much!

LuciferYang · 2023-01-31T18:50:47Z

It should not be released to the central repository yet. We need to wait for a while

dongjoon-hyun · 2023-01-31T18:51:06Z

Yes, it seems to be. I'll wait~

[ERROR] Failed to execute goal on project spark-core_2.12: Could not resolve dependencies for project org.apache.spark:spark-core_2.12:jar:spark-185468:
Could not find artifact org.xerial.snappy:snappy-java:jar:1.1.9.1 in gcs-maven-central-mirror (https://maven-central.storage-download.googleapis.com/maven2/) -> [Help 1]

dongjoon-hyun · 2023-02-01T05:37:27Z

Thank you again, @viirya .

dongjoon-hyun · 2023-02-01T06:01:53Z

Thank you, @LuciferYang .

### What changes were proposed in this pull request? This PR aims to upgrade `snappy-java` to 1.1.9.1. ### Why are the changes needed? This version has the following bug fixes. - Use original compressed and uncompressed buffer's position (xerial/snappy-java#293) - Fixed running snappy-java as OSGi bundle on Apple Silicon (M1 Pro) (xerial/snappy-java#303) - Avoid explicit class name in throw_exception (xerial/snappy-java#291) Here is the full release note. - https://github.com/xerial/snappy-java/releases/tag/v1.1.9.0 - https://github.com/xerial/snappy-java/releases/tag/v1.1.9.1 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes #39811 from dongjoon-hyun/SPARK-42242. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 65a1c16) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2023-02-01T06:15:41Z

All tests passed. Merged to master/3.4.
Thank you, @viirya , @HyukjinKwon , @LuciferYang , @srowen .

### What changes were proposed in this pull request? This PR aims to upgrade `snappy-java` to 1.1.9.1. ### Why are the changes needed? This version has the following bug fixes. - Use original compressed and uncompressed buffer's position (xerial/snappy-java#293) - Fixed running snappy-java as OSGi bundle on Apple Silicon (M1 Pro) (xerial/snappy-java#303) - Avoid explicit class name in throw_exception (xerial/snappy-java#291) Here is the full release note. - https://github.com/xerial/snappy-java/releases/tag/v1.1.9.0 - https://github.com/xerial/snappy-java/releases/tag/v1.1.9.1 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes apache#39811 from dongjoon-hyun/SPARK-42242. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 65a1c16) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions Bot added the BUILD label Jan 30, 2023

dongjoon-hyun commented Jan 30, 2023

View reviewed changes

viirya approved these changes Jan 30, 2023

View reviewed changes

dongjoon-hyun marked this pull request as draft January 31, 2023 00:06

dongjoon-hyun commented Jan 31, 2023

View reviewed changes

HyukjinKwon approved these changes Jan 31, 2023

View reviewed changes

LuciferYang approved these changes Jan 31, 2023

View reviewed changes

dongjoon-hyun added 2 commits January 31, 2023 19:31

[SPARK-42242][BUILD] Upgrade snappy-java to 1.1.9.0

9fe02c6

Use 1.1.9.1

e6cee58

dongjoon-hyun changed the title ~~[SPARK-42242][BUILD] Upgrade snappy-java to 1.1.9.0~~ [SPARK-42242][BUILD] Upgrade snappy-java to 1.1.9.1 Feb 1, 2023

dongjoon-hyun marked this pull request as ready for review February 1, 2023 03:36

viirya approved these changes Feb 1, 2023

View reviewed changes

LuciferYang approved these changes Feb 1, 2023

View reviewed changes

dongjoon-hyun closed this in 65a1c16 Feb 1, 2023

dongjoon-hyun deleted the SPARK-42242 branch February 1, 2023 06:15

Conversation

dongjoon-hyun commented Jan 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

viirya commented Jan 30, 2023

Uh oh!

dongjoon-hyun commented Jan 30, 2023

Uh oh!

dongjoon-hyun commented Jan 31, 2023

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 31, 2023

Uh oh!

LuciferYang left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Jan 31, 2023

Uh oh!

dongjoon-hyun commented Jan 31, 2023

Uh oh!

srowen commented Jan 31, 2023

Uh oh!

dongjoon-hyun commented Jan 31, 2023

Uh oh!

LuciferYang commented Jan 31, 2023

Uh oh!

dongjoon-hyun commented Jan 31, 2023

Uh oh!

LuciferYang commented Jan 31, 2023

Uh oh!

dongjoon-hyun commented Jan 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Feb 1, 2023

Uh oh!

dongjoon-hyun commented Feb 1, 2023

Uh oh!

dongjoon-hyun commented Feb 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dongjoon-hyun commented Jan 30, 2023 •

edited

Loading

dongjoon-hyun commented Jan 31, 2023 •

edited

Loading