repository·main·Indexed 18 days ago
https://github.com/albfernandez/juniversalchardetA Java port of Mozilla's 'universalchardet' library used for detecting the character encoding of text data. It provides tools for detecting encoding from byte streams or files via UniversalDetector and ReaderFactory, supporting a wide range of Chinese, Cyrillic, Greek, Hebrew, Japanese, Korean, and Unicode encodings.
To use juniversalchardet in your Java project, add the following dependency to your build configuration.
Maven
Add this to your pom.xml:
Gradle
Add this to your build.gradle:
<!-- Maven -->
<dependency>
<groupId>com.github.albfernandez</groupId>
<artifactId>juniversalchardet</artifactId>
<version>2.5.0</version>
</dependency>
<!-- Gradle -->
implementation 'com.github.albfernandez:juniversalchardet:2.5.0'For a quick way to detect the encoding of a specific java.io.File, use the static method UniversalDetector.detectCharset(File file). This method returns the encoding name as a String, or null if no encoding could be detected.
import org.mozilla.universalchardet.UniversalDetector;
public class TestDetectorFile {
public static void main (String[] args) throws java.io.IOException {
if (args.length != 1) {
System.err.println("Usage: java TestDetectorFile FILENAME");
System.exit(1);
}
java.io.File file = new java.io.File(args[0]);
String encoding = UniversalDetector.detectCharset(file);
if (encoding != null) {
System.out.println("Detected encoding = " + encoding);
} else {
System.out.println("No encoding detected.");
}
}
}To detect the encoding of a byte stream, follow these steps using org.mozilla.universalchardet.UniversalDetector:
UniversalDetector.handleData(byte[] buf, int offset, int length) repeatedly with chunks of data (typically a few thousand bytes).dataEnd() once all data has been processed.getDetectedCharset() to get the name of the detected encoding as a String.reset() if you intend to reuse the same detector instance for a different detection task.Note: You can check isDone() during the data feeding loop to stop early if the detector has already reached a confident conclusion.
import org.mozilla.universalchardet.UniversalDetector;
public class TestDetector
{
public static void main(String[] args)
{
byte[] buf = new byte[4096];
java.io.InputStream fis = java.nio.file.Files.newInputStream(java.nio.file.Paths.get("test.txt"));
// (1) Construct instance
UniversalDetector detector = new UniversalDetector();
// (2) Feed data
int nread;
while ((nread = fis.read(buf)) > 0 && !detector.isDone()) {
detector.handleData(buf, 0, nread);
}
// (3) Notify end of data
detector.dataEnd();
// (4) Get detected encoding
String encoding = detector.getDetectedCharset();
if (encoding != null) {
System.out.println("Detected encoding = " + encoding);
} else {
System.out.println("No encoding detected.");
}
// (5) Reset for reuse
detector.reset();
}
}Use org.mozilla.universalchardet.ReaderFactory to automatically detect a file's encoding and return a java.io.BufferedReader configured with that encoding. This abstracts away the manual detection and reader creation steps.
import org.mozilla.universalchardet.ReaderFactory;
public class TestCreateReaderFromFile {
public static void main (String[] args) throws java.io.IOException {
if (args.length != 1) {
System.err.println("Usage: java TestCreateReaderFromFile FILENAME");
System.exit(1);
}
java.io.Reader reader = null;
try {
java.io.File file = new java.io.File(args[0]);
reader = ReaderFactory.createBufferedReader(file);
// Do whatever you want with the reader
}
finally {
if (reader != null) {
reader.close();
}
}
}
}Ensure your environment meets the following requirements based on the version of juniversalchardet you are using:
Java (JDK)
Android
The library supports a wide range of encodings across several language groups. A full list of supported encodings is available in org.mozilla.universalchardet.Constants.
Supported Groups: