LCU14 BURLINGAME
Behan Webster, LCU14
LCU14-209: LLVMLinux
● About the LLVMLinux project● State of Linux kernel being compiled with clang● Details of specific known problems/patches still being upstreamed● How can you help?● Contact info
Linaro Overview
● Fully build the Linux kernel for multiple architectures, using the Clang/LLVM toolchain
● Discover LLVM/Kernel issues early and find fixes quickly across both communities
● Upstream patches to the Linux Kernel and LLVM projects
● Bring together like-minded developers● Enable the kernel community to do more in depth
analysis of the kernel code
The LLVMLinux Project Goals
● Fetches, patches, builds, tests: clang, kernel, qemu, etc○ git clone http://git.linuxfoundation.org/llvmlinux.git○ cd llvmlinux/target/vexpress (or vexpress64)○ make
LLVMLinux Build/Test System
● A mainline kernel tree with all LLVMLinux patches applied on top is now available:○ git://git.linuxfoundation.org/llvmlinux/kernel.git
● Dated llvmlinux branches○ remotes/origin/llvmlinux-2014.09.16
● The master branch is rebased regularly
Patched Mainline Kernel Tree
● LLVM/clang:○ All LLVMLinux patches for LLVM are Upstream○ Newer LLVM patches to support the Linux kernel are mostly
being added by upstream maintainers● Linux Kernel:
○ Roughly 49 kernel patches for various arches○ LLVMLinux branch in linux-next
LLVMLinux Project Status
● Patches still working their way upstream
Remaining LLVMLinux Kernel Patches
Architecture Number of Patches
Patches Submitted
Patches Accepted
all 23 17 2
arm 12 11 1 (+7)
aarch64 11 8 7
x86_64 3 1 1
TOTAL 49 37 11
● Total patches currently required for a single architecture
LLVMLinux Kernel Patches
Architecture Number of Patches
Patches Submitted
Patches Accepted
arm (+all) 35 (12+23) 28 (11+17) 3 (1+2)
aarch64 (+all) 34 (11+23) 25 (8+17) 9 (7+2)
x86_64 (+all) 26 (3+23) 18 (1+17) 3 (1+2)
● Basic Kbuild support for clang and x86 has been upstreamed● Specific support for ARM and aarch64 ready to be upstreamed
Kbuild
● Renato Golin, Vicinius Tinti, Saleem Abdulrasool and Stepan Dyatkovskiy are working on fixing IA issues in clang to support the Linux ARM kernel code (and ultimately AARCH64)
● David Woodhouse has added .code16 support for X86 ASM● For now we disable the IA and use GNU as instead
Integrated Assembly Status
● gcc passes -march to GNU as● clang doesn't... (Bug submitted PR)● Probably should be fixed in clang● Work around patch for now
-CFLAGS_aes-ce-cipher.o += -march=armv8-a+crypto
+CFLAGS_aes-ce-cipher.o += -march=armv8-a+crypto -Wa,-march=armv8-a+crypto
Different option passing
● GNU89/GNU90 (used by gcc)○ Function will be inlined where it is used○ No function definition is emitted○ A non-inlined function may also be provided
● GNU99/C99 (used by clang)○ Function will be inlined where it is used○ An external function is emitted○ No other function of the same name may be provided.
● Solution? Use “static inline” instead.● Only still an issue for ARM support for ftrace (submitted)
extern inline: Different for gnu89 and gnu99
● gcc is less picky about placement of __attribute__(())● clang requires it at the end of the type or variable
-struct __read_mostly va_alignment va_align = {
+struct va_alignment __read_mostly va_align = {
● (This particular patch was just accepted)
Attribute Order
● (Named registers for X86 kernel have been removed from the mainline kernel by Andi Kleen)
● ARM and AARCH64 still like using named registers● Clang now supports using a globally named register for the stack
pointer (Thanks Renato!)● For ARM/AARCH64 move to using a global in asm/thread_info.h
register unsigned long current_stack_pointer asm ("sp");
● Patches for AARCH64 now accepted, acked for ARM
Named Registers
● One of the uses of Named Registers in the ARM code is due to a deficiency in gcc
● The new code which works with gcc fails in clang● Solution, provide routines for both, and choose at compile time
● Gcc:asm("mrc p15, 0, %0, c13, c0, 4" : "=r" (off) : "Q" (*sp));
● Clang:asm("mrc p15, 0, %0, c13, c0, 4" : "=r" (off) : : "memory");
ARM percpu patch
● The following error is generated with clang for AArch64:error: invalid operand in inline asm: 'prfm pldl1keep, ${0:a}'
● Per comments by Tim Northover on the LLVM Bug database:It's rather unclear how it's better than "prfm pstl1keep, [%0]" though. Not all instructions can make use of any offset, so wouldn't we have to be conservative and always map it to "[xN]"?
● When %a0 is changed to [%x0] it uncovered a GCC bug: https://bugs.linaro.org/show_bug.cgi?id=635
● Changing the "p" to "r" resolves the issue for both clang and GCC.- asm volatile("prfm pldl1keep, %a0\n" : : "p" (ptr));+ asm volatile("prfm pldl1keep, [%x0]\n" : : "r" (ptr));
Missing “%a” for inline ASM
● By default clang merges globals with internal linkage into one: MergedGlobals
● Allows globals to be addressed using offsets from a base pointer● Can reduce the number of registers used● Modpost script in the Linux kernel uses symbol names to look for
section mismatches (e.g. regular code calling init code)● MergedGlobals breaks modpost (false positive section
mismatches)● Current solution: use -mno-global-merge to stop global merging● Updates to modpost may allow this optimization to be enabled
again in the future
Section Mismatch Issues (MergedGlobals)
● Clang emits code which uses the “aeabi” ARM calls which are implemented in compiler-rt (equivalient to libgcc)
● Compiler-rt doesn't easily cross compile yet...
void __aeabi_memcpy(void *dest, const void *src, size_t n)
void __aeabi_memmove(void *dest, const void *src, size_t n)
void __aeabi_memset(void *s, size_t n, int c)
● Still needed for ARM● No longer required for AARCH64
ARM eabi support
● VLAIS isn't supported by Clang (undocumented gcc extension) char vla[n]; /* Supported, C99/C11 */
struct {
char flexible_member[]; /* Supported, C99/C11 */
} struct_with_flexible_member;
struct {
char vlais[n]; /* Explicitly not allowed by C99/C11 */
} variable_length_array_in_struct;
● VLAIS is used in the Linux kernel in a number of places, spreading mostly through reusing patterns from data structures found in crypto
Variable Length Arrays In Structs
- struct {
- struct shash_desc shash;
- char ctx[crypto_shash_descsize(hash)];
- } desc;
+ char desc[sizeof(struct shash_desc)
+
+ crypto_shash_descsize(hash)] CRYPTO_MINALIGN_ATTR;
+ struct shash_desc *shash = (struct shash_desc *)desc;
unsigned int i;
- desc.shash.tfm = hash;
+ shash->tfm = hash;
VLAIS Removal Example (from crypto/hmac.c)
#define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long)
#define CRYPTO_MINALIGN ARCH_KMALLOC_MINALIGN
#define CRYPTO_MINALIGN_ATTR __attribute__ ((__aligned__(CRYPTO_MINALIGN)))
struct shash_desc {
struct crypto_shash *tfm;
u32 flags;
void *__ctx[] CRYPTO_MINALIGN_ATTR;
};
VLAIS Removal Example (the missing pieces)
● USB Gadget patch is in mainline● Mac80211 patch is in mainline● Netfilter patch is in mainline● apparmor patch accepted● Bluetooth patch accepted● Only the VLAIS patches for crypto are left:
(btrfs, dm-crypt, hmac, libcrc32c, testmgr, etc)
● However, we recently found a few previously unknown instances of VLAIS in raid10 and exofs...
Status of VLAIS in the Linux Kernel
static int _sp2d_alloc(unsigned pages_in_unit, unsigned group_width,
unsigned parity, struct __stripe_pages_2d **psp2d)
[...]
unsigned data_devs = group_width - parity;
struct _alloc_all_bytes {
struct __alloc_stripe_pages_2d {
struct __stripe_pages_2d sp2d;
struct __1_page_stripe _1p_stripes[pages_in_unit];
} __asp2d;
struct __alloc_1p_arrays {
struct page *pages[group_width];
struct page *scribble[group_width];
char page_is_read[data_devs];
} __a1pa[pages_in_unit];
} *_aab;
Huge use of VLAIS in fs/exofs/ore_raid.c
● Don’t use the non-C99 practices we showed in previous slides● Make it known you want to be able to use Clang to compile the
kernel (tell your Linaro representative!)● Test LLVMLinux patches● Report bugs to the LLVMLinux mailing list● Help get LLVMLinux patches upstream● Work on unsupported features and Bugs
○ http://llvm.linuxfoundation.org/index.php/Broken_kernel_options● Submit new targets and arch support● Patches welcome
How Can You Help?
Embrace theDragon.
He's cuddly.Thank you
http://llvm.linuxfoundation.org
● Project wiki page○ http://llvm.linuxfoundation.org
● Project Mailing List○ http://lists.linuxfoundation.org/mailman/listinfo/llvmlinux○ http://lists.linuxfoundation.org/pipermail/llvmlinux/
● IRC Channel○ #llvmlinux on OFTC○ http://buildbot.llvm.linuxfoundation.org/irclogs/OFTC/%23llvmlinux/
● LLVMLinux Community on Google Plus
Contribute to the LLVMLinux Project
More about Linaro Connect: connect.linaro.org Linaro members: www.linaro.org/membersMore about Linaro: www.linaro.org/about/