This work proposes heart rate variability (HRV) and kernel enhanced 1D-CNN based feature fusion technique for automatic anxiety detection from single channel wearable electrocardiogram (ECG) sensor signals. From the ECG sensor signals R-peaks are detected employing non-linear energy operator to extract efficient HRV features. To extract the inherent temporal features from the ECG signals a set of pre-initialized filters have been incorporated in a kernel-enhanced convolutional neural network model (1D-KCNN). Fused features are used with some standard classifiers to identify normal, light, moderate and severe anxiety levels. The performance of the algorithm for detection of anxiety is evaluated on a publicly available wearable ECG sensor dataset. This work has achieved 97.85% accuracy with 97.49% F1-score for detection of four anxiety classes using cross gradient boosting (XGBoost) model from short duration ECG signals. This work has outperformed recently published works on anxiety detection from ECG and multi-modal physiological signals.