Overview Statistic: PDF-Downloads (blue) and Frontdoor-Views (gray)

Transcoding unicode characters with AVX-512 instructions

  • Intel includes in its recent processors a powerful set of instructions capable of processing 512-bit registers with a single instruction (AVX-512). Some of these instructions have no equivalent in earlier instruction sets. We leverage these instructions to efficiently transcode strings between the most common formats: UTF-8 and UTF-16. With our novel algorithms, we are often twice as fast as the previous best solutions. For example, we transcode Chinese text from UTF-8 to UTF-16 at more than 5 GiB s−1 using fewer than 2 CPU instructions per character. To ensure reproducibility, we make our software freely available as an open-source library. Our library is part of the popular Node.js JavaScript runtime.

Export metadata

Additional Services

Share in Twitter Search Google Scholar Statistics - number of accesses to the document
Metadaten
Author:Robert Clausecker, Daniel Lemire
Document Type:Article
Parent Title (English):Software: Practice and Experience
Volume:53
Issue:12
First Page:2430
Last Page:2462
Publisher:John Wiley & Sons Ltd.
Year of first publication:2023
DOI:https://doi.org/10.1002/spe.3261
Accept ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.