Error: jar dependency conflicts

Example: Shading Java Classes Using Maven to address dependency conflicts 

When working with projects with multiple dependencies, there may be cases of conflicting dependencies between different versions of the same dependency within Qubole and in the custom jar used for the jobs.

For example, project A depends on project B and project C.2.0 and project B depends on project C.3.0. Usually both projects will download the jars C.2.0 and C.3.0 respectively and put them in their classpath. The problem may happen if C.2.0 and C.3.0 are not compatible. Project A, due to the dependencies of project B, will require both C.2.0 and C.3.0 in its classpath and then we will get conflicting jars resulting in errors.

The problem arises because project B (which is a dependency for project A), requires C.3.0 and it is not actually required by project A. In such cases, Maven allows a project "shade" certain packages/classes, that is, it will change the classpath of those packages and bundle it with the project.

Lets consider it with a real world example.

Hadoop2 project depends on guava-11.0, and wanted to integrate with ASCM (Qubole's auto scaling module). ASCM depends on guava-16.0 which is incompatible with guava-11.0. Hence running hadoop2-with-ascm will result in conflicting jars and hence errors.

We recommend using maven shading to solve this issue.

So, ASCM will modify its build process to "shade" all the jars represented by guava-16.0 and add it into its own package (ascm-1.0.jar) with a modified namespace. The modified pom.xml will look like this:

 <build>

  <plugins>
    <plugin>
      ...
    </plugin>
    <plugin>
      ...
    </plugin>
 
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-shade-plugin</artifactId>
      <version>2.3</version>
      <executions>
        <execution>
          <phase>package</phase>
          <goals>
            <goal>shade</goal>
          </goals>
          <configuration>
            <artifactSet>
              <excludes>
                <exclude>classworlds:classworlds</exclude>
                <exclude>javax.servlet:*</exclude>
                <exclude>commons-logging:commons-logging</exclude>
                <exclude>com.fasterxml.jackson.core:*</exclude>
                <exclude>logkit:*</exclude>
                <exclude>avalon-framework:*</exclude>
                <exclude>org.apache.maven:lib:tests</exclude>
                <exclude>log4j:log4j</exclude>
              </excludes>
            </artifactSet>
            <relocations>
              <relocation>
                <pattern>com.google</pattern>
                <shadedPattern>com.qubole.shaded.google</shadedPattern>
              </relocation>
            </relocations>
          </configuration>
        </execution>
      </executions>
    </plugin>
 
    <plugin>
      ...
    </plugin>
  </plugins>
</build>
 
 
Have more questions? Submit a request

Comments

  • Avatar
    Daniel Nuriyev

    I have a similar problem with kafka. Qubole comes with kafka 0.8 jars whereas I bundle my job with 0.10.2. The class loader that runs my job gives preference to Qubole's jars.

Powered by Zendesk