Disentangling elements of digital art

By Alistair Jones

SMU Office of Research – There is a sketch on the office wall of Shengfeng He, an Associate Professor of Computer Science at Singapore Management University (SMU). It evokes the style of a Japanese cartoon (anime) and depicts a princely warrior about to wield a sword. 

Could it be that the professor, who leads the Visual Understanding and Generation Laboratory at SMU, spent some of his youth fighting mock battles on a computer screen? 

"Yes, I’ve been a gamer since I was five and my love of drawing and anime naturally led me to explore digital art," he says. 

"It feels like the perfect fusion of my passions. The idea of turning my childhood hobbies into serious research is exciting, kind of like beating a final boss in a game, but for science."

Professor He is developing a tool called DreamDrawing, which utilises generative artificial intelligence (AI) to offer a new creative flexibility for digital artists.

"The proposal tackles the tricky issue of disentangling style and identity in character art," he says. "AI models can generate lots of visuals, but they tend to mix up a character’s defining features when you change its style. 

"DreamDrawing aims to let users manipulate artistic styles, whether it’s watercolour, pencil sketches or oil paintings, without losing the character’s core identity, ensuring that characters remain recognisable even when their visual style changes dramatically."

Deeply interconnected

So, why is it so difficult to disentangle and independently manipulate style and identity within the same image? 

"Style and identity are like two sides of the same coin in digital art; they are deeply interconnected. When you change a character’s style, such as turning their colours from vibrant to muted, it often shifts other aspects of their appearance, making it hard to recognise them," Professor He says. 

"Think of it as trying to put a superhero in a different costume without accidentally changing their powers. The challenge is keeping these elements separate but connected in just the right way."

Style can be an undetermined concept. Surely, there must be an issue in the training of large-scale text-to-image models to recognise it?

"Absolutely," Professor He says. "Style is like trying to describe the taste of a dish; everyone experiences it a little differently. Training AI to recognise and replicate an artistic style is challenging because there’s no fixed definition of what a style 'should' be. 

"Different artists may interpret the same style in unique ways. DreamDrawing helps tackle this by offering more precise controls over style elements, allowing the AI to capture the essence of a style without getting lost in subjective details."

To achieve this, Dream Drawing introduces special tokens that act like style and identity labels for the AI.

"We train these together using pre-trained diffusion models, which are great at refining noisy or messy images into something clear," Professor He says.

"By teaching the AI to separate these elements, we make sure it can swap styles without touching the key features that make a character who they are. So, Goku [from Dragon Ball] stays Goku, even if he’s rendered in watercolour."

Level of customisation

DreamDrawing’s innovation comes from its ability to independently control both style and identity using advanced AI techniques.

"It’s like having an AI artist that knows exactly how to balance creativity and consistency. This level of customisation is something that other systems haven’t quite nailed yet," Professor He says.

"DreamDrawing isn’t limited to cartoons and flat art styles. It’s capable of handling complex artistic techniques such as brush strokes, chiaroscuro and deep perspective. Imagine taking the Mona Lisa and recreating her with anime-style eyes or adding the dramatic lighting of a Baroque masterpiece. 

"DreamDrawing can do it. It’s designed to blend the nuances of traditional painting with digital flair, so it’s not just surface-level manipulation."

Another feature of DreamDrawing is an ability to render an image as a sketch, similar to a pencil drawing. And it, too, is not limited to character art.

"It can take any photograph and apply a sketch style while maintaining the subject’s identity," Professor He says.

"What’s exciting is that we also have another ongoing project focused on fine-grained sketch style, which can represent different stroke types, so you can expect a high level of detail and variation in the sketches. This makes our approach more versatile and expressive than many existing solutions."

Enhanced expression

The DreamDrawing model is still evolving.

"We’ve demonstrated the effectiveness of our techniques in style and identity disentanglement, and these early results give us confidence that we’re on the right track," Professor He says.

The potential for wider usages of DreamDrawing, beyond digital art-makers, spring to mind.

"Visitors [to art galleries] could reimagine historical pieces in different artistic styles or experiment with how various cultures might interpret the same subject.

"Museums could use this technology to offer workshops or digital exhibits, allowing people to create their own interpretations of famous works, fostering deeper engagement and learning," Professor He says.

"And DreamDrawing can enhance art education by giving students hands-on tools to explore and understand different artistic techniques. For example, students could use the platform to experiment with applying Renaissance painting styles to their own digital creations, or learn how different brush strokes can change the emotion of an artwork.

"This kind of interactive learning can help students grasp abstract concepts in art history and technique through direct experimentation."

DreamDrawing builds on Professor He's previous work in computational art and generative models. 

"For example, our collaboration with [Chinese technology conglomerate] Tencent to develop a real-time pixel art converter showed the potential of AI in the creative process. 

"DreamDrawing takes that further by focusing on the separation of style and identity, allowing more creative control for artists. It’s a natural next step in exploring how AI can enhance artistic expression without losing the essence of what makes each piece unique," Professor He says. 

Back to Research@SMU November 2024 Issue