This paper investigates the dynamic relationship between Japanese Yen exchange rates and market anxiety during the period from January 5, 1998 to April 18, 2016. A quantitative technique of multifractal detrended cross-correlation analysis (MF-DCCA) is used to explore the multifractal features of the cross-correlations between USD/JPY, AUD/JPY exchange rates and the market anxiety gauge VIX. The investigation shows that the causal relationship between Japanese Yen exchange rates and VIX are bidirectional in general, and the cross-correlations between the two sets of time series are multifractal. Strong evidence suggests that the cross-correlation exponents tend to exhibit different volatility patterns in response to diverse external shocks such as financial distress and widening in interest rate spread, suggesting that the cross-correlated behavior between Japanese Yen exchange rates and VIX are susceptible to economic uncertainties and risks. In addition, the performances of two market anxiety gauges, the VIX and the TED spread, are compared and the sources of multifractality are also traced. Thus, this paper contributes to the literature by shedding light on the unique driving forces of the Yen exchange rate fluctuations in the international foreign exchange market.