RGBT tracking usually suffers from various challenging factors of fast motion, scale variation, illumination variation,thermal crossover and occlusion, to name a few. Existing works often study fusion models to solve all challenges simultaneously, which requires fusion models complex enough and training data large enough, and are usually diffcult to be constructed in real-world scenarios. In this work, we disentangle the fusion process via the challenge attributes, and thus propose a novel Attribute-Based Progressive Fusion Network (APFNet) to increase the fusion capacity with a small number of parameters while reducing the dependence on large-scale training data. In particular, we design fve attribute-specifc fusion branches to integrate RGB and thermal features under the challenges of thermal crossover, illumination variation, scale variation, occlusion and fast motion respectively. By disentangling the fusion process, we can use a small number of parameters for each branch to achieve robust fusion of different modalities and train each branch using the small training subset with the corresponding attribute annotation. Then, to adaptive fuse features of all branches, we design an aggregation fusion module based on SKNet. Finally, we also design an enhancement fusion transformer to strengthen the aggregated feature and modality-specifc features. Experimental results on benchmark datasets demonstrate the effectiveness of our APFNet against other state-of-the-art methods.